Near eye diffractive holographic projection method

ABSTRACT

An augmented reality display device (such as a head mounted device) includes a partially transparent and partially reflective lens, a laser light source, a radio frequency source, a display controller, an acousto-optical modulator, and a microelectromechanical (MEMS) device. The laser light source generates light. The radio frequency (RF) source generates a RF signal. The display controller generates a synchronization signal. The acousto-optical modulator receives at least a portion of the light, modulates the light based on the RF signal, and provides modulated light. The MEMS device receives the synchronization signal from the display controller and reflects the modulated light towards the partially transparent and partially reflective lens. The MEMS device determines a direction in which the modulated light reflects based on the synchronization signal and the partially transparent and partially reflective lens reflecting the modulated laser light towards an eye of a user of the augmented realty display device.

RELATED APPLICATION

This application claims the benefit of priority of U.S. ProvisionalApplication No. 62/355,112, filed Jun. 27, 2016, which is hereinincorporated by reference in its entirety.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to an augmentedreality display device. Specifically, the present disclosure addressessystems and methods for a transparent parabolic reflector of anaugmented reality device.

BACKGROUND

Holography enables three-dimensional (3D) images to be recorded in anoptical medium for later reconstruction and display. Typically, ahologram is constructed by optical interference of two coherent laserbeams in a film or a grating. As such, the laser recording impartsstatic optical properties such as fixed depth encoded lights in thegrating. The characteristics of the grating do not change once therecording is performed. As such, static optical properties of gratingscan be difficult to use in Augmented Reality (AR) devices since theuser's relative position is dynamic. AR devices allow users to observe ascene while simultaneously seeing relevant virtual content that may bealigned to items, images, objects, or environments in the field of viewof the device or user. However, the user may move the devices relativeto the items and stationary objects in space. Since the depth of fieldfor the virtual content is fixed based on the recorded grating, the usermay perceive a disparity between the real object and the virtualcontent.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 is a block diagram illustrating an example of a networkenvironment suitable for an augmented reality user interface, accordingto some example embodiments.

FIG. 2 is a block diagram illustrating an example embodiment ofcomponents of a head mounted device.

FIG. 3 is a block diagram illustrating an example embodiment of adisplay controller.

FIG. 4 is a flow diagram illustrating a method of forming a head mounteddevice in accordance with one embodiment.

FIG. 5 is a flow diagram illustrating a method of operating a headmounted device in accordance with one embodiment.

FIG. 6 a block diagram illustrating components of a machine, accordingto some example embodiments, able to read instructions from amachine-readable medium and perform any one or more of the methodologiesdiscussed herein.

FIG. 7 is a block diagram illustrating an example embodiment of anoperation of a head mounted device.

FIG. 8 is a block diagram illustrating another example embodiment of anoperation of a head mounted device.

DETAILED DESCRIPTION

Example methods and systems are directed an augmented reality displaydevice including a transparent parabolic reflector. Examples merelytypify possible variations. Unless explicitly stated otherwise,structures (e.g., structural components, such as modules) are optionaland may be combined or subdivided, and operations (e.g., in a procedure,algorithm, or other function) may vary in sequence or be combined orsubdivided. In the following description, for purposes of explanation,numerous specific details are set forth to provide a thoroughunderstanding of example embodiments. It will be evident to one skilledin the art, however, that the present subject matter may be practicedwithout these specific details.

FIG. 1 is a block diagram illustrating an example of a networkenvironment suitable for an augmented reality user interface, accordingto some example embodiments. A network environment 100 includes a headmounted device 102 and a server 110, communicatively coupled to eachother via a network 108. The head mounted device 102 and the server 110may each be implemented in a computer system, in whole or in part, asdescribed below with respect to FIG. 6.

The server 110 may be part of a network-based system. For example, thenetwork-based system may be or include a cloud-based server system thatprovides additional information, such as 3D models or other virtualobjects, to the head mounted device 102.

A user 106 may wear the head mounted device 102 and look at a physicalobject 104 in a real world physical environment. The user 106 may be ahuman user (e.g., a human being), a machine user (e.g., a computerconfigured by a software program to interact with the head mounteddevice 102), or any suitable combination thereof (e.g., a human assistedby a machine or a machine supervised by a human). The user 106 is notpart of the network environment 100, but is associated with the headmounted device 102. For example, the head mounted device 102 may be acomputing device with a camera and a transparent display such as atablet, smartphone, or a wearable computing device (e.g., helmet orglasses). In another example embodiment, the computing device may behand held or may be removably mounted to the head of the user 106. Inone example, the display may be a screen that displays what is capturedwith a camera of the head mounted device 102. In another example, thedisplay of the head mounted device 102 may be transparent orsemi-transparent such as in lenses of wearable computing glasses or thevisor or a face shield of a helmet.

The user 106 may be a user of an AR application in the head mounteddevice 102 and at the server 110. The AR application may provide theuser 102 with an AR experience triggered by identified objects (e.g.,physical object 104) in the physical environment. For example, thephysical object 104 include identifiable objects such as a 2D physicalobject (e.g., a picture), a 3D physical object (e.g., a factorymachine), a location (e.g., at the bottom floor of a factory), or anyreferences (e.g., perceived corners of walls or furniture) in the realworld physical environment. The AR application may include computervision recognition to determine corners, objects, lines, letters, etc.

In one example embodiment, the objects in the image are tracked andrecognized locally in the head mounted device 102 using a local contextrecognition dataset or any other previously stored dataset of the ARapplication of the head mounted device 102. The local contextrecognition dataset module may include a library of virtual objectsassociated with real-world physical objects or references. In oneexample, the head mounted device 102 identifies feature points in animage of the physical object 104. The head mounted device 102 may alsoidentify tracking data related to the physical object 104 (e.g., GPSlocation of the head mounted device 102, orientation, distance to thephysical object 104). If the captured image is not recognized locally atthe head mounted device 102, the head mounted device 102 can downloadadditional information (e.g., 3D model or other augmented data)corresponding to the captured image, from a database of the server 110over the network 108.

In another example embodiment, the physical object 104 in the image istracked and recognized remotely at the server 110 using a remote contextrecognition dataset or any other previously stored dataset of an ARapplication in the server 110. The remote context recognition datasetmodule may include a library of virtual objects or augmented informationassociated with real-world physical objects or references.

External sensors 112 may be associated with, coupled to, related to thephysical object 104 to measure a location, status, and characteristicsof the physical object 104. Examples of measured readings may includeand but are not limited to weight, pressure, temperature, velocity,direction, position, intrinsic and extrinsic properties, acceleration,and dimensions. For example, external sensors 112 may be disposedthroughout a factory floor to measure movement, pressure, orientation,and temperature. The external sensors 112 can also be used to measure alocation, status, and characteristics of the head mounted device 102 andthe user 106. The server 110 can compute readings from data generated bythe external sensors 112. The server 110 can generate virtual indicatorssuch as vectors or colors based on data from external sensors 112.Virtual indicators are then overlaid on top of a live image or a view ofthe physical object 104 in a line of sight of the user 106 to show datarelated to the object 116. For example, the virtual indicators mayinclude arrows with shapes and colors that change based on real-timedata. The visualization may be provided to the physical object 104 sothat the head mounted device 102 can render the virtual indicators in adisplay of the head mounted device 102. In another example embodiment,the virtual indicators are rendered at the server 110 and streamed tothe head mounted device 102.

The external sensors 112 may include other sensors used to track thelocation, movement, and orientation of the head mounted device 102externally without having to rely on sensors internal to the headmounted device 102. The sensors 112 may include optical sensors (e.g.,depth-enabled 3D camera), wireless sensors (Bluetooth, Wi-Fi), GPSsensors, and audio sensors to determine the location of the user 106wearing the head mounted device 102, distance of the user 106 to theexternal sensors 112 (e.g., sensors placed in corners of a venue or aroom), the orientation of the head mounted device 102 to track what theuser 106 is looking at (e.g., direction at which the head mounted device102 is pointed, e.g., head mounted device 102 pointed towards a playeron a tennis court, head mounted device 102 pointed at a person in aroom).

In another example embodiment, data from the external sensors 112 andinternal sensors in the head mounted device 102 may be used foranalytics data processing at the server 110 (or another server) foranalysis on usage and how the user 106 is interacting with the physicalobject 104 in the physical environment. Live data from other servers mayalso be used in the analytics data processing. For example, theanalytics data may track at what locations (e.g., points or features) onthe physical or virtual object the user 106 has looked, how long theuser 106 has looked at each location on the physical or virtual object,how the user 106 wore the head mounted device 102 when looking at thephysical or virtual object, which features of the virtual object theuser 106 interacted with (e.g., such as whether the user 106 engagedwith the virtual object), and any suitable combination thereof. The headmounted device 102 receives a visualization content dataset related tothe analytics data. The head mounted device 102 then generates a virtualobject with additional or visualization features, or a new experience,based on the visualization content dataset.

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. For example, a computer system able to implement any one or moreof the methodologies described herein is discussed below with respect toFIG. 19. As used herein, a “database” is a data storage resource and maystore data structured as a text file, a table, a spreadsheet, arelational database (e.g., an object-relational database), a triplestore, a hierarchical data store, or any suitable combination thereof.Moreover, any two or more of the machines, databases, or devicesillustrated in FIG. 1 may be combined into a single machine, and thefunctions described herein for any single machine, database, or devicemay be subdivided among multiple machines, databases, or devices.

The network 108 may be any network that enables communication between oramong machines (e.g., server 110), databases, and devices (e.g., device101). Accordingly, the network 108 may be a wired network, a wirelessnetwork (e.g., a mobile or cellular network), or any suitablecombination thereof. The network 108 may include one or more portionsthat constitute a private network, a public network (e.g., theInternet), or any suitable combination thereof.

FIG. 2 is a block diagram illustrating an example embodiment of modules(e.g., components) of the head mounted device 102. The head mounteddevice 102 (e.g., a helmet, a visor, or any other device mounted to ahead for the user 106) includes an augmented reality device 202 and alens 220. In one example, the lens 220 transmits light coming from thephysical object 104 but also reflects light coming from a MEM device226. The lens 220 may include, for example, a visor or transparent faceshield positioned in front of the eyes of the user 106. In one exampleembodiment, the lens 220 includes a partially transparent parabolicreflector 222 or mirror. The partially transparent parabolic reflector222 includes a convex side and a concave side. A reflective coatingmaterial, such as silver, may be applied to the concave side of thepartially transparent parabolic reflector 222 to enable the lens 220 toreflect light from the augmented reality device 202 towards an eye 218of the user 106. In another example embodiment, the partiallytransparent parabolic reflector 222 has a focus point positioned at oraround the eye 218.

The augmented reality device 202 includes sensors 204, a processor 210,a display controller 206, an acousto-optical modulator (AOM) 224, a(Micro-Electro Mechanical) MEMS device 226, and a storage device 208.The augmented reality device 202 may generate a light signal projectedonto the lens 220. The lens 220 combines light reflected from theaugmented reality device 202 with light reflected off the physicalobject 104 towards the eye 218 of the user 106.

The sensors 204 include, for example, a thermometer, an infrared camera,a barometer, a humidity sensor, an EEG (electroencephalogram) sensor, aproximity or location sensor (e.g, near field communication, globalpositioning system (GPS), Bluetooth, Wifi), an optical sensor (e.g.,camera), an orientation sensor (e.g., gyroscope), an audio sensor (e.g.,a microphone), or any suitable combination thereof. For example, thesensors 204 may include a rear facing camera and a front facing camerain the augmented reality device 202. It is noted that the types ofsensors described herein are for illustration purposes and the sensors204 are thus not limited to the ones described.

The processor 210 includes an AR application 212, a rendering module214, and a dynamic depth encoder 216. The AR application 212 receivesdata from sensors 204 (e.g., receive an image of the physical object104) and identifies and recognizes the physical object 104 usingmachine-vision recognition techniques. The AR application 212 thenretrieves, from the storage device 208, AR content associated with thephysical object 104. In one example embodiment, the AR application 212identifies a visual reference (e.g., a logo or quick response (QR) code)on the physical object (e.g., a chair) and tracks the visual reference.The visual reference may also be referred to as a marker and may consistof an identifiable image, symbol, letter, number, machine-readable code.For example, the visual reference may include a bar code, a QR code, oran image that has been previously associated with the virtual object.

The rendering module 214 renders virtual objects based on data fromsensors 204. For example, the rendering module 214 renders a display ofa virtual object (e.g., a door with a color based on the temperatureinside the room as detected by sensors from head mounted displays (HMDs)inside the room) based on a three-dimensional model of the virtualobject (e.g., 3D model of a virtual door) associated with the physicalobject 104 (e.g., a physical door). In another example, the renderingmodule 214 generates a display of the virtual object overlaid on animage of the physical object 104 captured by a camera of the augmentedreality device 202. The virtual object may be further manipulated (e.g.,by the user 106) by moving the physical object 104 relative to theaugmented reality device 202. Similarly, the display of the virtualobject may be manipulated (e.g., by the user 104) by moving theaugmented reality device 202 relative to the physical object 104.

In one example embodiment, the rendering module 214 identifies thephysical object 112 (e.g., a physical telephone) based on data fromsensors 204 and external sensors 110, accesses virtual functions (e.g.,increase or lower the volume of a nearby television) associated withphysical manipulations (e.g., lifting a physical telephone handset) ofthe physical object 104, and generates a virtual function correspondingto a physical manipulation of the physical object 104.

In another example embodiment, the rendering module 214 determineswhether the captured image matches an image locally stored in thestorage device 208 that includes a local database of images andcorresponding additional information (e.g., three-dimensional model andinteractive features). The rendering module 214 retrieves a primarycontent dataset from the server 108, generates and updates a contextualcontent dataset based on an image captured with the augmented realitydevice 202.

The dynamic depth encoder 216 determines depth information of thevirtual content based on the depth of the content or portion of thecontent relative to the lens 220. The depth information is provided tothe display controller 206.

The display controller 206 includes, for example, a hardware such as agraphics card configured to generate images based on the data signals(e.g., content information and depth information) provided by theprocessor 210. In one example embodiment, the display controller 206includes an RF source and a laser source. The display controller 206 isdescribed in more details with respect to FIG. 3. The display controller206 utilizes the depth information from the dynamic depth encoder 216 togenerate the RF signal which drives the acousto-optical modulator 224.The generated surface acoustic wave in an optical element of theacousto-optical modulator 224 alters the diffraction of light throughthe optical element to produce a holographic image with the associateddepth of field information of the content. Through acousto-opticmodulation, light can be modulated through the optical element at a highrate (e.g., frequency) so that the user does not perceive individualchanges in the depth of field. In another example, the dynamic depthencoder 216 adjusts the depth of field based on sensor data from thesensors 204. For example, the depth of field may be increased based onthe distance between the lens 220 and the physical object 104. Inanother example, the depth of field may be adjusted based on a directionin which the eyes are looking.

In one example embodiment, the display controller 206 generates asynchronization signal to both the acousto-optical modulator 224 and theMEMS device 226. The synchronization signal enables the MEMS device 226to operate and reflect corresponding individual light beams from theacousto-optical modulator 224.

The MEMS device 226 receives a projection of the holographic image(e.g., light signal) from the acousto-optical modulator 224 and reflectscorresponding individual light beams to the partially transparentparabolic reflector 222. The MEMS device 226 reflects the light beamsbased on the synchronization signal from the display controller 206 toguide the corresponding individual light beams to the correspondinglocations on the partially transparent parabolic reflector 222. The MEMSdevice 226 includes, for example, one or more mirrors. The position andorientation of the mirrors is controlled and adjusted based on thesynchronization signal received from the display controller 206.

The reflector of the lens provides a near-eye focusing and reflectionmechanism to project AR contents into the eye. The MEMS device issuggested so that subsets of the full image bouncing off each individualmirror in the MEMS array can be adjusted in timing, intensities,projection directions, appeared distance. etc. The optical path of thelight ray is so small (10˜20 cm, tens of nanosecond range) that the timedifference of travel off each mirror is negligible, i.e. they are allsync'ed within a tolerable timing range to a grand video trigger signalgenerated by the AR device. All the parts to the full image in the eyeare correctly stitched. Therefore, the pitch/yaw angles of the MEMSmirrors are to follow a monotonically increasing or decreasing orderalong each dimension.

Once the current positions of the light modulator 224, the reflector228, the MEMS 226, and the eye 218 are obtained via feedback of theirplacement motors or monitoring cameras, the trajectory of the main beamis defined. Then a variation angle range for the MEMS mirror iscalculated, to avoid sweeping the AR content outside the span of thereflector. This range is obtained for both sweeping angles of themirror, i.e. pitch for horizontal and yaw for vertical.

The partially transparent parabolic reflector 222 reflects the lightsignals from the MEMS device 226 to the eye 218. Aluminum is usuallyapplied by vapor deposition of thickness on the ˜100 nm scale with 50%transparency. In one example embodiment, this aluminum coating can beapplied on the concave side facing the user eye of lens 220. To avoidthe vignette (“spyglass”) effect, the aluminum coating is kept away fromthe edge of lens 220.

FIG. 7 is a block diagram illustrating an example embodiment of anoperation of a head mounted device. FIG. 8 is a block diagramillustrating another example embodiment of an operation of a headmounted device.

The focus could potentially be adjusted by one or a combination of thefollowing schemes:

1. Adjust the relative distance between the modulator 224 and the MEMSdevice 226.2. Steering the mirrors in the MEMS device 226.3. Move the lens 220 towards or away from the user eye.4. Slightly stretch or bend the lens 220 (e.g. by piezoelectric motors)to change the curvature of the reflector so its focal length. Thestorage device 208 stores an identification of the sensors and theirrespective functions. The storage device 208 further includes a databaseof visual references (e.g., images, visual identifiers, features ofimages) and corresponding experiences (e.g., three-dimensional virtualobjects, interactive features of the three-dimensional virtual objects).For example, the visual reference may include a machine-readable code ora previously identified image (e.g., a picture of shoe). The previouslyidentified image of the shoe may correspond to a three-dimensionalvirtual model of the shoe that can be viewed from different angles bymanipulating the position of the HMD 102 relative to the picture of theshoe. Features of the three-dimensional virtual shoe may includeselectable icons on the three-dimensional virtual model of the shoe. Anicon may be selected or activated using a user interface on the HMD 102.

In another example embodiment, the storage device 208 includes a primarycontent dataset, a contextual content dataset, and a visualizationcontent dataset. The primary content dataset includes, for example, afirst set of images and corresponding experiences (e.g., interactionwith three-dimensional virtual object models). For example, an image maybe associated with one or more virtual object models. The primarycontent dataset may include a core set of images of the most popularimages determined by the server 110. The core set of images may includea limited number of images identified by the server 110. For example,the core set of images may include the images depicting covers of theten most popular magazines and their corresponding experiences (e.g.,virtual objects that represent the ten most popular magazines). Inanother example, the server 110 may generate the first set of imagesbased on the most popular or often scanned images received at the server110. Thus, the primary content dataset does not depend on objects orimages scanned by the rendering module 214 of the HMD 102.

The contextual content dataset includes, for example, a second set ofimages and corresponding experiences (e.g., three-dimensional virtualobject models) retrieved from the server 110. For example, imagescaptured with the HMD 102 that are not recognized (e.g., by the server110) in the primary content dataset are submitted to the server 110 forrecognition. If the captured image is recognized by the server 110, acorresponding experience may be downloaded at the HMD 102 and stored inthe contextual content dataset. Thus, the contextual content datasetrelies on the context in which the HMD 102 has been used. As such, thecontextual content dataset depends on objects or images scanned by therendering module 214 of the HMD 102.

In one embodiment, the HMD 102 may communicate over the network 108 withthe server 110 to retrieve a portion of a database of visual references,corresponding three-dimensional virtual objects, and correspondinginteractive features of the three-dimensional virtual objects. Thenetwork 108 may be any network that enables communication between oramong machines, databases, and devices (e.g., the HMD 102). Accordingly,the network 108 may be a wired network, a wireless network (e.g., amobile or cellular network), or any suitable combination thereof. Thenetwork 108 may include one or more portions that constitute a privatenetwork, a public network (e.g., the Internet), or any suitablecombination thereof.

FIG. 3 illustrates an aspect of the subject matter in accordance withone embodiment. The display controller 206 may be implemented as a videosignal generator having various components for generating the datasignals communicated to the acousto-optical modulator 224. The displaycontroller 206 includes one or more RF amplifiers 302, an RF mixer 304,one or more laser source 306, a graphics processing unit (GPU) 314, aclock sync 308, and an output interface 310. The GPU 314 renders acomputer-generated-holographic (CGH) pattern for forming a hologram inthe acousto-optical modulator 224. The RF mixer 304 mixes the outputfrom the GPU 314 with a predetermined RF signal. The RF amplifiers 302then amplify the mixed GPU output and RF signal. In an alternativeembodiment, the RF amplifiers 302 amplify the RF energy before it ismixed with the GPU output. The laser source 306 generate a laser lightthat will be used to illuminate the hologram generated in theacousto-optical modulator 224. The clock sync 308 synchronizes the lightoutput and the mixed RF signal, which is then output via the outputinterface 310 and to the acousto-optical modulator 224.

FIG. 4 is a flow diagram illustrating a method 400 of forming a headmounted device in accordance with one embodiment. At operation 404, aparabolic reflector or mirror is disposed or formed inside a lens of ahead mounted device. At operation 406, a MEMS device 226 is embedded inthe head mounted device. For example, the MEMS device 226 may bedisposed above the eye of the user when wearing the head mounted device.In another example, the MEMS device 226 may be disposed adjacent to thesides of the eyes of the user in the head mounted device. For the sakeof simplifying the optics layout and using less mirrors, the MEMS devicefaces both the output of the acousto-optical modulator and the near eyelens/reflector, concave side. It could be near either the modulator orthe lens, as long as the bounced hologram light has a clear path hittingthe parabolic reflector. Also see Diagram 1 attached. At operation 408,the acousto-optical modulator 224 projects light signal to the MEMSdevice 226.

At operation 410, the MEMS device 226 reflects the projected lightsignal to the parabolic reflector 222. The MEMS device 226 guides theindividual beam in a direction based on the synchronization signal fromthe display controller 206.

FIG. 5 is a flow diagram illustrating a method 500 of operating a headmounted device in accordance with one embodiment. The method 500 isimplemented by one or more of the components 302-314 of the displaycontroller 206, the acousto-optical modulator 224, and the MEMS device226 and is discussed by way of reference thereto.

Initially, at operation 504, the display controller 206 generates the RFsignals and the laser light associated with the content to be displayedto the user.

At 506, the display controller 206 sends the RF signal and the laserlight to the acousto-optical modulator 224. The acousto-opticalmodulator 224 includes an optical element whose optical properties canbe modulated by the RF signal, to form the desired holographic patternnecessary to generate a hologram when illuminated by the laser light.

At operation 508, the display controller 206 sends a synchronizationsignal to the MEMS device 226.

At operation 510, the acousto-optical modulator 224 modulates the laserlight to allow for generation of images of hologram based on the RFsignal from the display controller 206.

At operation 512, the acousto-optical modulator 224 projects themodulated light to the MEMS device 226.

At operation 514, the MEMS device 226 reflects the beam of AOM-modulatedlight rapidly based on the synchronization signal from displaycontroller 206, to a half-silvered parabolic lens (e.g., parabolicreflector 222). The holographic light is reflected off the concave sideof the half-silvered parabolic lens into the user's eye 218. Externallight is able to pass through from the convex side of the half-silveredparabolic lens to the user's eye 218. The reflected light and theexternal light are combined at the user's eye 218.

FIG. 6 is a block diagram illustrating components of a machine 600,according to some example embodiments, able to read instructions 606from a computer-readable medium 618 (e.g., a non-transitorymachine-readable medium, a machine-readable storage medium, acomputer-readable storage medium, or any suitable combination thereof)and perform any one or more of the methodologies discussed herein, inwhole or in part. Specifically, the machine 600 in the example form of acomputer system (e.g., a computer) within which the instructions 606(e.g., software, a program, an application, an applet, an app, or otherexecutable code) for causing the machine 600 to perform any one or moreof the methodologies discussed herein may be executed, in whole or inpart.

In alternative embodiments, the machine 600 operates as a standalonedevice or may be communicatively coupled (e.g., networked) to othermachines. In a networked deployment, the machine 600 may operate in thecapacity of a server machine or a client machine in a server-clientnetwork environment, or as a peer machine in a distributed (e.g.,peer-to-peer) network environment. The machine 600 may be a servercomputer, a client computer, a personal computer (PC), a tabletcomputer, a laptop computer, a netbook, a cellular telephone, asmartphone, a set-top box (STB), a personal digital assistant (PDA), aweb appliance, a network router, a network switch, a network bridge, orany machine capable of executing the instructions 606, sequentially orotherwise, that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute the instructions 606 to perform all or part of any oneor more of the methodologies discussed herein.

The machine 600 includes a processor 604 (e.g., a CPU, a graphicsprocessing unit (GPU), a digital signal processor (DSP), an ASIC, aradio-frequency integrated circuit (RFIC), or any suitable combinationthereof), a main memory 610, and a static memory 622, which areconfigured to communicate with each other via a bus 612. The processor604 contains solid-state digital microcircuits (e.g., electronic,optical, or both) that are configurable, temporarily or permanently, bysome or all of the instructions 606 such that the processor 604 isconfigurable to perform any one or more of the methodologies describedherein, in whole or in part. For example, a set of one or moremicrocircuits of the processor 604 may be configurable to execute one ormore modules (e.g., software modules) described herein. In some exampleembodiments, the processor 604 is a multicore CPU (e.g., a dual-coreCPU, a quad-core CPU, or a 128-core CPU) within which each of multiplecores behaves as a separate processor that is able to perform any one ormore of the methodologies discussed herein, in whole or in part.Although the beneficial effects described herein may be provided by themachine 600 with at least the processor 604, these same beneficialeffects may be provided by a different kind of machine that contains noprocessors (e.g., a purely mechanical system, a purely hydraulic system,or a hybrid mechanical-hydraulic system), if such a processor-lessmachine is configured to perform one or more of the methodologiesdescribed herein.

The machine 600 may further include a video display 608 (e.g., a plasmadisplay panel (PDP), a light emitting diode (LED) display, a liquidcrystal display (LCD), a projector, a cathode ray tube (CRT), or anyother display capable of displaying graphics or video). The machine 600may also include an alpha-numeric input device 614 (e.g., a keyboard orkeypad), a cursor control device 616 (e.g., a mouse, a touchpad, atrackball, a joystick, a motion sensor, an eye tracking device, or otherpointing instrument), a drive unit 602, a signal generation device 620(e.g., a sound card, an amplifier, a speaker, a headphone jack, or anysuitable combination thereof), and a network interface device 624.

The drive unit 602 (e.g., a data storage device) includes thecomputer-readable medium 618 (e.g., a tangible and non-transitorymachine-readable storage medium) on which are stored the instructions606 embodying any one or more of the methodologies or functionsdescribed herein. The instructions 606 may also reside, completely or atleast partially, within the main memory 610, within the processor 604(e.g., within the processor's cache memory), or both, before or duringexecution thereof by the machine 600. Accordingly, the main memory 610and the processor 604 may be considered machine-readable media (e.g.,tangible and non-transitory machine-readable media). The instructions606 may be transmitted or received over a computer network 626 via thenetwork interface device 624. For example, the network interface device624 may communicate the instructions 606 using any one or more transferprotocols (e.g., hypertext transfer protocol (HTTP)).

In some example embodiments, the machine 600 may be a portable computingdevice (e.g., a smart phone, tablet computer, or a wearable device), andhave one or more additional input components (e.g., sensors or gauges).Examples of such input components include an image input component(e.g., one or more cameras), an audio input component (e.g., one or moremicrophones), a direction input component (e.g., a compass), a locationinput component (e.g., a GPS receiver), an orientation component (e.g.,a gyroscope), a motion detection component (e.g., one or moreaccelerometers), an altitude detection component (e.g., an altimeter), abiometric input component (e.g., a heartrate detector or a bloodpressure detector), and a gas detection component (e.g., a gas sensor).Input data gathered by any one or more of these input components may beaccessible and available for use by any of the modules described herein.

As used herein, the term “memory” refers to a machine-readable mediumable to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While thecomputer-readable medium 618 is shown in an example embodiment to be asingle medium, the term “machine-readable medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storeinstructions. The term “machine-readable medium” shall also be taken toinclude any medium, or combination of multiple media, that is capable ofstoring the instructions 606 for execution by the machine 600, such thatthe instructions 606, when executed by one or more processors of themachine 600 (e.g., processor 604), cause the machine 600 to perform anyone or more of the methodologies described herein, in whole or in part.Accordingly, a “machine-readable medium” refers to a single storageapparatus or device, as well as cloud-based storage systems or storagenetworks that include multiple storage apparatus or devices. The term“machine-readable medium” shall accordingly be taken to include, but notbe limited to, one or more tangible and non-transitory data repositories(e.g., data volumes) in the example form of a solid-state memory chip,an optical disc, a magnetic disc, or any suitable combination thereof. A“non-transitory” machine-readable medium, as used herein, specificallydoes not include propagating signals per se. In some exampleembodiments, the instructions 606 for execution by the machine 600 maybe communicated by a carrier medium. Examples of such a carrier mediuminclude a storage medium (e.g., a non-transitory machine-readablestorage medium, such as a solid-state memory, being physically movedfrom one place to another place) and a transient medium (e.g., apropagating signal that communicates the instructions 606).

Certain example embodiments are described herein as including modules.Modules may constitute software modules (e.g., code stored or otherwiseembodied in a machine-readable medium or in a transmission medium),hardware modules, or any suitable combination thereof. A “hardwaremodule” is a tangible (e.g., non-transitory) physical component (e.g., aset of one or more processors) capable of performing certain operationsand may be configured or arranged in a certain physical manner. Invarious example embodiments, one or more computer systems or one or morehardware modules thereof may be configured by software (e.g., anapplication or portion thereof) as a hardware module that operates toperform operations described herein for that module.

In some example embodiments, a hardware module may be implementedmechanically, electronically, hydraulically, or any suitable combinationthereof. For example, a hardware module may include dedicated circuitryor logic that is permanently configured to perform certain operations. Ahardware module may be or include a special-purpose processor, such as afield programmable gate array (FPGA) or an ASIC. A hardware module mayalso include programmable logic or circuitry that is temporarilyconfigured by software to perform certain operations. As an example, ahardware module may include software encompassed within a CPU or otherprogrammable processor. It will be appreciated that the decision toimplement a hardware module mechanically, hydraulically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity that may be physically constructed,permanently configured (e.g., hardwired), or temporarily configured(e.g., programmed) to operate in a certain manner or to perform certainoperations described herein. Furthermore, as used herein, the phrase“hardware-implemented module” refers to a hardware module. Consideringexample embodiments in which hardware modules are temporarily configured(e.g., programmed), each of the hardware modules need not be configuredor instantiated at any one instance in time. For example, where ahardware module includes a CPU configured by software to become aspecial-purpose processor, the CPU may be configured as respectivelydifferent special-purpose processors (e.g., each included in a differenthardware module) at different times. Software (e.g., a software module)may accordingly configure one or more processors, for example, to becomeor otherwise constitute a particular hardware module at one instance oftime and to become or otherwise constitute a different hardware moduleat a different instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over suitable circuits and buses)between or among two or more of the hardware modules. In embodiments inwhich multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module may perform an operation and store theoutput of that operation in a memory (e.g., a memory device) to which itis communicatively coupled. A further hardware module may then, at alater time, access the memory to retrieve and process the stored output.Hardware modules may also initiate communications with input or outputdevices, and can operate on a resource (e.g., a collection ofinformation from a computing resource).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module in which the hardware includes one or more processors.Accordingly, the operations described herein may be at least partiallyprocessor-implemented, hardware-implemented, or both, since a processoris an example of hardware, and at least some operations within any oneor more of the methods discussed herein may be performed by one or moreprocessor-implemented modules, hardware-implemented modules, or anysuitable combination thereof.

Moreover, such one or more processors may perform operations in a “cloudcomputing” environment or as a service (e.g., within a “software as aservice” (SaaS) implementation). For example, at least some operationswithin any one or more of the methods discussed herein may be performedby a group of computers (e.g., as examples of machines that includeprocessors), with these operations being accessible via a network (e.g.,the Internet) and via one or more appropriate interfaces (e.g., anapplication program interface (API)). The performance of certainoperations may be distributed among the one or more processors, whetherresiding only within a single machine or deployed across a number ofmachines. In some example embodiments, the one or more processors orhardware modules (e.g., processor-implemented modules) may be located ina single geographic location (e.g., within a home environment, an officeenvironment, or a server farm). In other example embodiments, the one ormore processors or hardware modules may be distributed across a numberof geographic locations.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures and theirfunctionality presented as separate components and functions in exampleconfigurations may be implemented as a combined structure or componentwith combined functions. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents and functions. These and other variations, modifications,additions, and improvements fall within the scope of the subject matterherein.

Some portions of the subject matter discussed herein may be presented interms of algorithms or symbolic representations of operations on datastored as bits or binary digital signals within a memory (e.g., acomputer memory or other machine memory). Such algorithms or symbolicrepresentations are examples of techniques used by those of ordinaryskill in the data processing arts to convey the substance of their workto others skilled in the art. As used herein, an “algorithm” is aself-consistent sequence of operations or similar processing leading toa desired result. In this context, algorithms and operations involvephysical manipulation of physical quantities. Typically, but notnecessarily, such quantities may take the form of electrical, magnetic,or optical signals capable of being stored, accessed, transferred,combined, compared, or otherwise manipulated by a machine. It isconvenient at times, principally for reasons of common usage, to referto such signals using words such as “data,” “content,” “bits,” “values,”“elements,” “symbols,” “characters,” “terms,” “numbers,” “numerals,” orthe like. These words, however, are merely convenient labels and are tobe associated with appropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “accessing,” “processing,” “detecting,” “computing,”“calculating,” “determining,” “generating,” “presenting,” “displaying,”or the like refer to actions or processes performable by a machine(e.g., a computer) that manipulates or transforms data represented asphysical (e.g., electronic, magnetic, or optical) quantities within oneor more memories (e.g., volatile memory, non-volatile memory, or anysuitable combination thereof), registers, or other machine componentsthat receive, store, transmit, or display information. Furthermore,unless specifically stated otherwise, the terms “a” or “an” are hereinused, as is common in patent documents, to include one or more than oneinstance. Finally, as used herein, the conjunction “or” refers to anon-exclusive “or,” unless specifically stated otherwise.

The following enumated embodiments describe various example embodimentsof methods, machine-readable media, and systems (e.g., machines,devices, or other apparatus) discussed herein.

A first embodiment provides an augmented reality display devicecomprising:

a partially transparent and partially reflective lens;a laser light source configured to generate light;a radio frequency source configured to generate a radio frequency (RF)signal;a display controller configured to generate a synchronization signal;an acousto-optical modulator configured to receive at least a portion ofthe light, modulate the light based on the radio frequency signal, andprovide modulated light; anda microelectromechanical (MEMS) device configured to receive thesynchronization signal from the display controller and reflect themodulated light towards the partially transparent reflective lens, theMEMS device determining a direction in which the modulated lightreflects based on the synchronization signal and the partiallytransparent and partially reflective lens reflecting the modulated laserlight towards an eye of a user of the augmented realty display device.

A second embodiment provides a device according to the first embodiment,wherein the partially transparent reflective lens comprises a paraboliclens including a concave side, a convex side, and a partially reflectivelayer applied to the concave side, wherein the parabolic lens forms afocus point located about the eye of the user.

A third embodiment provides a device according to the first embodiment,wherein the partially transparent and partially reflective lens combinesan external light reflected off a physical object with the modulatedlaser light at the eye of the user

A second embodiment provides a device according to the first embodiment,further comprising:

one or more processors comprising an augmented reality applicationconfigured to generate augmented reality content and to send a datasignal of the augmented reality content to the display controller.

A fifth embodiment provides a device according to the fourth embodiment,wherein the augmented reality application comprises:

a recognition module configured to identify a physical object imaged bya camera of the augmented reality display device;a rendering module configured to retrieve the augmented reality contentassociated with the physical object; anda dynamic depth encoder configured to identify a depth of the physicalobject and encode a depth of the augmented reality content into the RFsignal, the depth of the augmented reality content being coded based onthe depth of the physical object.

A sixth embodiment provides a device according to the fifth embodiment,wherein the display controller comprises:

a graphics processing unit (GPU) configured to render an image of theaugmented reality content and generate a GPU output signal correspondingto the rendered image;an RF source configured to generate the RF signal;an RF mixer configured to mix the GPU output signal with the RF signal;anda laser source configured to generate the laser light based on the mixedGPU output signal.

A seventh embodiment provides a device according to the firstembodiment, wherein the display controller comprises:

a graphics processing unit (GPU) configured to render an image of anaugmented reality content and generate a GPU output signal correspondingto the rendered image, the augmented reality content independent of datafrom a camera of the augmented reality display device;an RF source configured to generate the RF signal;an RF mixer configured to mix the GPU output signal with the RF signal;anda laser source configured to generate the laser light based on the mixedGPU output signal.

An eight embodiment provides a device according to the first embodiment,wherein the display controller comprises:

a graphics processing unit (GPU) configured to render an image of anaugmented reality content and generate a GPU output signal correspondingto the rendered image, a depth of the rendered image independent of datafrom a camera of the augmented reality display device;an RF source configured to generate the RF signal;an RF mixer configured to mix the GPU output signal with the RF signal;anda laser source configured to generate the laser light based on the mixedGPU output signal.

A ninth embodiment provides a device according to the first embodiment,wherein the MEMS device is synchronized with the display controller viaa reference timing, the reference timing further synchronizing thedisplay controller with the acousto-optical modulator.

A tenth embodiment provides a device according to the first embodiment,wherein the MEMS device reflects and spreads a single beam of themodulated laser light in different directions based on the RF signaltowards the partially transparent reflective lens.

What is claimed is:
 1. An augmented reality display device comprising: a partially transparent and partially reflective lens; a laser light source configured to generate light; a radio frequency source configured to generate a radio frequency (RF) signal; a display controller configured to generate a synchronization signal; an acousto-optical modulator configured to receive at least a portion of the light, modulate the light based on the radio frequency signal, and provide modulated light; and a microelectromechanical (MEMS) device configured to receive the synchronization signal from the display controller and reflect the modulated light towards the partially transparent reflective lens, the MEMS device determining a direction in which the modulated light reflects based on the synchronization signal and the partially transparent and partially reflective lens reflecting the modulated laser light towards an eye of a user of the augmented realty display device.
 2. The augmented reality display device of claim 1, wherein the partially transparent reflective lens comprises a parabolic lens including a concave side, a convex side, and a partially reflective layer applied to the concave side, wherein the parabolic lens forms a focus point located about the eye of the user.
 3. The augmented reality display device of claim 1, wherein the partially transparent and partially reflective lens combines an external light reflected off a physical object with the modulated laser light at the eye of the user
 4. The augmented reality display device of claim 1, further comprising: one or more processors comprising an augmented reality application configured to generate augmented reality content and to send a data signal of the augmented reality content to the display controller.
 5. The augmented reality display device of claim 4, wherein the augmented reality application comprises: a recognition module configured to identify a physical object imaged by a camera of the augmented reality display device; a rendering module configured to retrieve the augmented reality content associated with the physical object; and a dynamic depth encoder configured to identify a depth of the physical object and encode a depth of the augmented reality content into the RF signal, the depth of the augmented reality content being coded based on the depth of the physical object.
 6. The augmented reality display device of claim 5, wherein the display controller comprises: a graphics processing unit (GPU) configured to render an image of the augmented reality content and generate a GPU output signal corresponding to the rendered image; an RF source configured to generate the RF signal; an RF mixer configured to mix the GPU output signal with the RF signal; and a laser source configured to generate the laser light based on the mixed GPU output signal.
 7. The augmented reality display device of claim 1, wherein the display controller comprises: a graphics processing unit (GPU) configured to render an image of an augmented reality content and generate a GPU output signal corresponding to the rendered image, the augmented reality content independent of data from a camera of the augmented reality display device; an RF source configured to generate the RF signal; an RF mixer configured to mix the GPU output signal with the RF signal; and a laser source configured to generate the laser light based on the mixed GPU output signal.
 8. The augmented reality display device of claim 1, wherein the display controller comprises: a graphics processing unit (GPU) configured to render an image of an augmented reality content and generate a GPU output signal corresponding to the rendered image, a depth of the rendered image independent of data from a camera of the augmented reality display device; an RF source configured to generate the RF signal; an RF mixer configured to mix the GPU output signal with the RF signal; and a laser source configured to generate the laser light based on the mixed GPU output signal.
 9. The augmented reality display device of claim 1, wherein the MEMS device is synchronized with the display controller via a reference timing, the reference timing further synchronizing the display controller with the acousto-optical modulator.
 10. The augmented reality display device of claim 1, wherein the MEMS device reflects and spreads a single beam of the modulated laser light in different directions based on the RF signal towards the partially transparent reflective lens.
 11. A method of forming a holographic image at a head mounted device, the method comprising: generating a radiofrequency (RF) signal, a laser light, and a synchronization signal at a display controller of the head mounted device; receiving the RF signal and the laser light at an acousto-optical modulator; modulating the laser light based on the RF signal at the acousto-optical modulator; projecting the modulated laser light to a microelectromechanical system (MEMS) device; providing the synchronization signal and the modulated laser light to the MEMS device; and directing the MEMS device to reflect the modulated laser light in a direction based on the synchronization signal towards a partially transparent and partially reflective lens of the head mounted device, the partially transparent reflective lens reflecting the modulated laser light towards an eye of a user of the head mounted device.
 12. The method of claim 11, wherein the partially transparent and partially reflective lens comprises a parabolic lens including a concave side, a convex side, and a reflective layer applied to the concave side, wherein the parabolic lens forms a focus point located about the eye of the user.
 13. The method of claim 11, wherein the partially transparent and partially reflective lens combines an external light reflected off a physical object with the modulated laser light at the eye of the user
 14. The method of claim 11, further comprising: generating an augmented reality content with an augmented reality application implemented in one or more processors in the head mounted device; sending data signal of the augmented reality content to the display controller.
 15. The method of claim 14, further comprising: identifying a physical object imaged by a camera of the augmented reality display device; retrieving the augmented reality content associated with the physical object; identifying a depth of the physical object; and encoding a depth of the augmented reality content into the RF signal, the depth of the augmented reality content being coded based on the depth of the physical object.
 16. The method of claim 15, further comprising: rendering an image of the augmented reality content with a Graphical Processing Unit (GPU); generating a GPU output signal corresponding to the rendered image; generating the RF signal with an RF source; mixing the GPU output signal with the RF signal using an RF mixer; and generating the laser light based on the mixed GPU output signal.
 17. The method of claim 11, further comprising: rendering an image of an augmented reality content with a GPU; generating a GPU output signal corresponding to the rendered image, the augmented reality content independent of data from a camera of the augmented reality display device; generating the RF signal with an RF source; mixing the GPU output signal with the RF signal using an RF mixer; and generating the laser light based on the mixed GPU output signal.
 18. The method of claim 11, further comprising: rendering an image of an augmented reality content with a GPU; generating a GPU output signal corresponding to the rendered image, a depth of the rendered image independent of data from a camera of the augmented reality display device; generating the RF signal with an RF source; mixing the GPU output signal with the RF signal using an RF mixer; and generating the laser light based on the mixed GPU output signal.
 19. The method of claim 11, further comprising: synchronizing the MEMS device with the display controller via a reference timing; and synchronizing the display controller with the acousto-optical modulator using the reference timing.
 20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that, when executed by a computer, cause the computer to: generate an RF signal, a laser light, and a synchronization signal at a display controller of the head mounted device; receive the RF signal and the laser light at an acousto-optical modulator; modulate the laser light based on the RF signal at the acousto-optical modulator; project the modulated laser light to a microelectromechanical system (MEMS) device; provide the synchronization signal and the modulated laser light to the MEMS device; and direct the MEMS device to reflect the modulated laser light in a direction based on the synchronization signal towards a partially transparent and partially reflective lens of the head mounted device, the partially transparent reflective lens reflecting the modulated laser light towards an eye of a user of the head mounted device. 